Noise Injection: Theoretical Prospects

Noise Injection consists in adding noise to the inputs during neural network training. Experimental results suggest that it might improve the generalization ability of the resulting neural network. A justiication of this improvement remains elusive: rst, describing analytically the average perturbed cost function is diicult, second, controlling the uctuations of the random perturbed cost function is hard. Hence recent papers suggest to replace the random perturbed cost by a (deterministic) Taylor approximation of the average perturbed cost function. This paper takes a diierent stance: when the injected noise is Gaussian, Noise Injection is naturally connected to the action of the Heat Kernel. This provides indications on the relevance domain of traditional Taylor expansions, and shows the dependence of the quality of Taylor approximations on global smoothness properties of neural networks under consideration. The connection between noise injection and Heat kernel also enables to control the uctuations of the random perturbed cost function. Under the previously mentioned global smoothness assumption, tools from Gaussian analysis provide bounds on the tail behavior of the perturbed cost. This nally suggests that mixing input perturbation with smoothness based penalization might be prootable.